Solving factored MDPs using non-homogeneous partitions
نویسندگان
چکیده
We present an algorithm for aggregating states in solving large MDPs (represented as factored MDPs) using search by successive re nement in the space of nonhomogeneous partitions. Homogeneity is de ned in terms of stochastic bisimulation and reward equivalence within blocks of a partition. Since homogeneous partitions that de ne equivalent reduced-state-space MDPs can have a large number of blocks, we relax the requirement of homogeneity. The algorithm constructs approximate aggregate MDPs from non-homogeneous partitions, solves the aggregate MDPs exactly, and then uses the resulting value functions as part of a heuristic in re ning the current best non-homogeneous partition. We outline the theory motivating the use of this heuristic and present empirical results. In addition to investigating more exhaustive local search methods we explore the use of techniques derived from research on discretizing continuous state spaces. Finally, we compare the results from our algorithms which search in the space of non-homogeneous partitions with exact and approximate algorithms which represent homogeneous and approximately homogeneous partitions as decision trees or algebraic decision diagrams.
منابع مشابه
Solving Factored MDPs via Non-Homogeneous Partitioning
This paper describes an algorithm for solving large state-space MDPs (represented as factored MDPs) using search by successive refinement in the space of non-homogeneous partitions. Homogeneity is defined in terms of bisimulation and reward equivalence within blocks of a partition. Since homogeneous partitions that define equivalent reduced state-space MDPs can have a large number of blocks, we...
متن کاملContext-specific Sign-propagation in Qualitative Probabilistic Networks
This paper describes an algorithm for solving large state-space MDPs (represented as factored MDPs) using search by successive refinement in the space of non-homogeneous partitions. Homogeneity is defined in terms of bisimulation and reward equivalence within blocks of a partition. Since homogeneous partitions that define equivalent reduced state-space MDPs can have a large number of blocks, we...
متن کاملModel Reduction Techniques for Computing
We present a method for solving implicit (factored) Markov decision processes (MDPs) with very large state spaces. We introduce a property of state space partitions which we call-homogeneity. Intuitively, an-homogeneous partition groups together states that behave approximately the same under all or some subset of policies. Borrowing from recent work on model minimization in computer-aided soft...
متن کاملModel Reduction Techniques for Computing ApproximatelyOptimal Solutions for Markov Decision
We present a method for solving implicit (factored) Markov decision processes (MDPs) with very large state spaces. We introduce a property of state space partitions which we call-homogeneity. Intuitively, an-homogeneous partition groups together states that behave approximately the same under all or some subset of policies. Borrowing from recent work on model minimization in computer-aided soft...
متن کاملModel Reduction Techniques for Computing Approximately Optimal Solutions for Markov Decision Processes
We present a method for solving implicit (factored) Markov decision processes (MDPs) with very large state spaces. We intro duce a property of state space partitions which we call f-homogeneity. Intuitively, an f-homogeneous partition groups together states that behave approximately the same under all or some subset of policies. Borrow ing from recent work on model minimization in computer-ai...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Artif. Intell.
دوره 147 شماره
صفحات -
تاریخ انتشار 2003